Originally published on Towards AI the World's Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale AI and technology startups. Let us help you unleash your technology to the masses. Data Engineers are at the heart of the engine room of any data-driven company.

According to Oracle, feature extraction is an attribute reduction process, which results in a much smaller and richer set of attributes. Depending on the requirements, identifying and extracting informative and compact data sets (for an ML model) may need structured data like numbers and dates or unstructured data like categorical features and raw text. If the data volume is large, the feature extraction can be handled separately, and the generated features can be stored in the storage layer. The format of the stored features is ready for direct consumption by the ML training process in the next phase. The feature extraction can be done for a wide range of applications like simple ETL process, model prediction pipeline, or retraining the model based on new data to improve the model accuracy.

As a consumer of goods and services, you experience the results of machine learning (ML) whenever the institutions you rely on use ML processes to run their operations. You may receive a text message from a bank requiring verification after the bank has paused a credit card transaction. Or, an online travel site may send you an email that offers personalized accommodations for your next personal or business trip. The work that happens behind the scenes to facilitate these experiences can be difficult to fully realize or appreciate. An important portion of that work is done by the data engineering teams that build the data pipelines to help train and deploy those ML models.

Data pipelines are where most of the time is spent for those working with data because the bulk of a machine learning project involves data collection and cleaning. Loominus gives everyone the power to build the data pipelines critical to any machine learning project. Teraport is a powerful tool within the Loominus product suite that ingests and stages data. In another post, we'll discuss the data ingestion APIs. We're going to build a data pipeline that generates the average credit score of borrowers within a portfolio of loans.